Skip to content

Update rook version from 1.12 to 1.13 #1210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nirs merged 1 commit into from
Feb 22, 2024
Merged

Update rook version from 1.12 to 1.13 #1210

nirs merged 1 commit into from
Feb 22, 2024

Conversation

@yati1998
Copy link

@yati1998 yati1998 commented Feb 19, 2024

This current rook version fails to take latest ci changes. This commit updates the rook version.

@yati1998
update rook version from 12 - 13
5d32433 
This current rook version fails to take latest ci changes.
This commit updates the rook version.

Signed-off-by: yati1998 <ypadia@redhat.com>
@yati1998
Copy link
Author

yati1998 commented Feb 20, 2024

#1205

@nirs nirs linked an issue Feb 20, 2024 that may be closed by this pull request
update rook version to v1.13 #1205
Closed
@nirs
Copy link
Member

nirs commented Feb 20, 2024

Thanks @yati1998! can you describe how did you test this? Did you start the envs/rook.yaml with this change? Did you start envs/regional-dr.yaml and run basic-test/run with this change?

@yati1998
Copy link
Author

yati1998 commented Feb 20, 2024

Thanks @yati1998! can you describe how did you test this? Did you start the envs/rook.yaml with this change? Did you start envs/regional-dr.yaml and run basic-test/run with this change?

I started the envs/rook.yaml with this change and see everything working fine, but didn't test it with regional-dr.yaml

@nirs
Copy link
Member

nirs commented Feb 20, 2024

Tried to test locally, so far starting regional dr failed because after we cleared
osd blocklist, new blocklist entries were added while we configured rbd mirroring.
We did not have this issue in 1.12.

$ drenv start envs/regional-dr.yaml 
2024-02-20 22:37:02,998 INFO    [rdr] Starting environment
2024-02-20 22:37:03,047 INFO    [dr1] Starting minikube cluster
2024-02-20 22:37:04,131 INFO    [dr2] Starting minikube cluster
2024-02-20 22:37:05,069 INFO    [hub] Starting minikube cluster
2024-02-20 22:39:11,669 INFO    [hub] Cluster started in 126.60 seconds
2024-02-20 22:39:11,670 INFO    [hub/0] Running addons/ocm-hub/start
2024-02-20 22:39:11,670 INFO    [hub/1] Running addons/submariner/start
2024-02-20 22:39:28,695 INFO    [dr1] Cluster started in 145.65 seconds
2024-02-20 22:39:28,695 INFO    [dr1/0] Running addons/cert-manager/start
2024-02-20 22:39:28,695 INFO    [dr1/1] Running addons/ocm-cluster/start
2024-02-20 22:39:28,696 INFO    [dr1/2] Running addons/csi-addons/start
2024-02-20 22:39:42,300 INFO    [dr1/0] addons/cert-manager/start completed in 13.61 seconds
2024-02-20 22:39:42,301 INFO    [dr1/0] Running addons/rook-operator/start
2024-02-20 22:39:51,490 INFO    [dr1/2] addons/csi-addons/start completed in 22.79 seconds
2024-02-20 22:39:51,490 INFO    [dr1/2] Running addons/olm/start
2024-02-20 22:39:59,000 INFO    [hub/0] addons/ocm-hub/start completed in 47.33 seconds
2024-02-20 22:39:59,000 INFO    [hub/0] Running addons/ocm-controller/start
2024-02-20 22:40:09,983 INFO    [hub/1] addons/submariner/start completed in 58.31 seconds
2024-02-20 22:40:09,984 INFO    [hub/1] Running addons/submariner/test
2024-02-20 22:40:25,966 INFO    [dr1/2] addons/olm/start completed in 34.48 seconds
2024-02-20 22:40:25,966 INFO    [dr1/2] Running addons/minio/start
2024-02-20 22:40:30,032 INFO    [hub/0] addons/ocm-controller/start completed in 31.03 seconds
2024-02-20 22:40:30,032 INFO    [hub/0] Running addons/cert-manager/start
2024-02-20 22:40:37,247 INFO    [dr2] Cluster started in 213.12 seconds
2024-02-20 22:40:37,247 INFO    [dr2/0] Running addons/cert-manager/start
2024-02-20 22:40:37,248 INFO    [dr2/1] Running addons/ocm-cluster/start
2024-02-20 22:40:37,248 INFO    [dr2/2] Running addons/csi-addons/start
2024-02-20 22:40:41,301 INFO    [dr1/2] addons/minio/start completed in 15.33 seconds
2024-02-20 22:40:41,301 INFO    [dr1/2] Running addons/velero/start
2024-02-20 22:40:44,338 INFO    [dr1/0] addons/rook-operator/start completed in 62.04 seconds
2024-02-20 22:40:44,338 INFO    [dr1/0] Running addons/rook-cluster/start
2024-02-20 22:40:49,465 INFO    [hub/0] addons/cert-manager/start completed in 19.43 seconds
2024-02-20 22:40:49,465 INFO    [hub/0] Running addons/olm/start
2024-02-20 22:40:56,393 INFO    [dr2/0] addons/cert-manager/start completed in 19.15 seconds
2024-02-20 22:40:56,393 INFO    [dr2/0] Running addons/rook-operator/start
2024-02-20 22:40:56,562 INFO    [hub/1] addons/submariner/test completed in 46.58 seconds
2024-02-20 22:41:00,016 INFO    [dr2/2] addons/csi-addons/start completed in 22.77 seconds
2024-02-20 22:41:00,016 INFO    [dr2/2] Running addons/olm/start
2024-02-20 22:41:18,115 INFO    [hub/0] addons/olm/start completed in 28.65 seconds
2024-02-20 22:41:24,847 INFO    [dr1/2] addons/velero/start completed in 43.55 seconds
2024-02-20 22:41:24,847 INFO    [dr1/2] Running addons/velero/test
2024-02-20 22:41:40,656 INFO    [dr2/2] addons/olm/start completed in 40.64 seconds
2024-02-20 22:41:40,656 INFO    [dr2/2] Running addons/minio/start
2024-02-20 22:41:56,751 INFO    [dr1/1] addons/ocm-cluster/start completed in 148.06 seconds
2024-02-20 22:41:56,751 INFO    [dr1/1] Running addons/ocm-cluster/test
2024-02-20 22:41:58,669 INFO    [dr2/2] addons/minio/start completed in 18.01 seconds
2024-02-20 22:41:58,669 INFO    [dr2/2] Running addons/velero/start
2024-02-20 22:42:00,808 INFO    [dr1/2] addons/velero/test completed in 35.96 seconds
2024-02-20 22:42:04,079 INFO    [dr1/1] addons/ocm-cluster/test completed in 7.33 seconds
2024-02-20 22:42:04,079 INFO    [dr1/1] Running addons/recipe/start
2024-02-20 22:42:05,524 INFO    [dr1/1] addons/recipe/start completed in 1.45 seconds
2024-02-20 22:42:09,151 INFO    [dr2/0] addons/rook-operator/start completed in 72.76 seconds
2024-02-20 22:42:09,151 INFO    [dr2/0] Running addons/rook-cluster/start
2024-02-20 22:42:14,482 INFO    [dr2/1] addons/ocm-cluster/start completed in 97.23 seconds
2024-02-20 22:42:14,483 INFO    [dr2/1] Running addons/ocm-cluster/test
2024-02-20 22:42:21,355 INFO    [dr2/1] addons/ocm-cluster/test completed in 6.87 seconds
2024-02-20 22:42:21,355 INFO    [dr2/1] Running addons/recipe/start
2024-02-20 22:42:23,769 INFO    [dr2/1] addons/recipe/start completed in 2.41 seconds
2024-02-20 22:42:28,877 INFO    [dr2/2] addons/velero/start completed in 30.21 seconds
2024-02-20 22:42:28,877 INFO    [dr2/2] Running addons/velero/test
2024-02-20 22:43:03,706 INFO    [dr2/2] addons/velero/test completed in 34.83 seconds
2024-02-20 22:43:19,945 INFO    [dr1/0] addons/rook-cluster/start completed in 155.61 seconds
2024-02-20 22:43:19,945 INFO    [dr1/0] Running addons/rook-pool/start
2024-02-20 22:43:37,179 INFO    [dr1/0] addons/rook-pool/start completed in 17.23 seconds
2024-02-20 22:43:37,180 INFO    [dr1/0] Running addons/rook-toolbox/start
2024-02-20 22:43:40,974 INFO    [dr1/0] addons/rook-toolbox/start completed in 3.79 seconds
2024-02-20 22:44:17,988 INFO    [dr2/0] addons/rook-cluster/start completed in 128.84 seconds
2024-02-20 22:44:17,988 INFO    [dr2/0] Running addons/rook-pool/start
2024-02-20 22:44:55,322 INFO    [dr2/0] addons/rook-pool/start completed in 37.33 seconds
2024-02-20 22:44:55,322 INFO    [dr2/0] Running addons/rook-toolbox/start
2024-02-20 22:44:58,185 INFO    [dr2/0] addons/rook-toolbox/start completed in 2.86 seconds
2024-02-20 22:44:58,185 INFO    [rdr/0] Running addons/rbd-mirror/start
2024-02-20 22:44:58,186 INFO    [rdr/1] Running addons/volsync/start
2024-02-20 22:45:31,319 INFO    [rdr/1] addons/volsync/start completed in 33.13 seconds
2024-02-20 22:45:31,319 INFO    [rdr/1] Running addons/volsync/test
2024-02-20 22:45:57,297 ERROR   [rdr/0] Cluster failed
Traceback (most recent call last):
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 129, in execute
    f.result()
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 289, in run_worker
    run_addon(addon, worker["name"], hooks=hooks, allow_failure=allow_failure)
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 305, in run_addon
    run_hook(hook, addon["args"], name, allow_failure=allow_failure)
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 312, in run_hook
    run(hook, *args, name=name)
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 327, in run
    for line in commands.watch(*cmd):
  File "/home/nsoffer/src/ramen/test/drenv/commands.py", line 148, in watch
    raise Error(args, error, exitcode=p.returncode)
drenv.commands.Error: Command failed:
   command: ('addons/rbd-mirror/start', 'dr1', 'dr2')
   exitcode: 1
   error:
      Traceback (most recent call last):
        File "/home/nsoffer/src/ramen/test/addons/rbd-mirror/start", line 175, in <module>
          check_blocklist(cluster2)
        File "/home/nsoffer/src/ramen/test/addons/rbd-mirror/start", line 37, in check_blocklist
          raise RuntimeError(f"Ceph blocklist on cluster {cluster}: {blocklist}")
      RuntimeError: Ceph blocklist on cluster dr2: [{'addr': '192.168.122.192:0/216466187', 'until': '2024-02-20T21:45:03.442684+0000'}]

2024-02-20 22:46:31,992 INFO    [rdr/1] addons/volsync/test completed in 60.67 seconds

@yati1998
Copy link
Author

yati1998 commented Feb 21, 2024

Tried to test locally, so far starting regional dr failed because after we cleared osd blocklist, new blocklist entries were added while we configured rbd mirroring. We did not have this issue in 1.12.

$ drenv start envs/regional-dr.yaml 
2024-02-20 22:37:02,998 INFO    [rdr] Starting environment
2024-02-20 22:37:03,047 INFO    [dr1] Starting minikube cluster
2024-02-20 22:37:04,131 INFO    [dr2] Starting minikube cluster
2024-02-20 22:37:05,069 INFO    [hub] Starting minikube cluster
2024-02-20 22:39:11,669 INFO    [hub] Cluster started in 126.60 seconds
2024-02-20 22:39:11,670 INFO    [hub/0] Running addons/ocm-hub/start
2024-02-20 22:39:11,670 INFO    [hub/1] Running addons/submariner/start
2024-02-20 22:39:28,695 INFO    [dr1] Cluster started in 145.65 seconds
2024-02-20 22:39:28,695 INFO    [dr1/0] Running addons/cert-manager/start
2024-02-20 22:39:28,695 INFO    [dr1/1] Running addons/ocm-cluster/start
2024-02-20 22:39:28,696 INFO    [dr1/2] Running addons/csi-addons/start
2024-02-20 22:39:42,300 INFO    [dr1/0] addons/cert-manager/start completed in 13.61 seconds
2024-02-20 22:39:42,301 INFO    [dr1/0] Running addons/rook-operator/start
2024-02-20 22:39:51,490 INFO    [dr1/2] addons/csi-addons/start completed in 22.79 seconds
2024-02-20 22:39:51,490 INFO    [dr1/2] Running addons/olm/start
2024-02-20 22:39:59,000 INFO    [hub/0] addons/ocm-hub/start completed in 47.33 seconds
2024-02-20 22:39:59,000 INFO    [hub/0] Running addons/ocm-controller/start
2024-02-20 22:40:09,983 INFO    [hub/1] addons/submariner/start completed in 58.31 seconds
2024-02-20 22:40:09,984 INFO    [hub/1] Running addons/submariner/test
2024-02-20 22:40:25,966 INFO    [dr1/2] addons/olm/start completed in 34.48 seconds
2024-02-20 22:40:25,966 INFO    [dr1/2] Running addons/minio/start
2024-02-20 22:40:30,032 INFO    [hub/0] addons/ocm-controller/start completed in 31.03 seconds
2024-02-20 22:40:30,032 INFO    [hub/0] Running addons/cert-manager/start
2024-02-20 22:40:37,247 INFO    [dr2] Cluster started in 213.12 seconds
2024-02-20 22:40:37,247 INFO    [dr2/0] Running addons/cert-manager/start
2024-02-20 22:40:37,248 INFO    [dr2/1] Running addons/ocm-cluster/start
2024-02-20 22:40:37,248 INFO    [dr2/2] Running addons/csi-addons/start
2024-02-20 22:40:41,301 INFO    [dr1/2] addons/minio/start completed in 15.33 seconds
2024-02-20 22:40:41,301 INFO    [dr1/2] Running addons/velero/start
2024-02-20 22:40:44,338 INFO    [dr1/0] addons/rook-operator/start completed in 62.04 seconds
2024-02-20 22:40:44,338 INFO    [dr1/0] Running addons/rook-cluster/start
2024-02-20 22:40:49,465 INFO    [hub/0] addons/cert-manager/start completed in 19.43 seconds
2024-02-20 22:40:49,465 INFO    [hub/0] Running addons/olm/start
2024-02-20 22:40:56,393 INFO    [dr2/0] addons/cert-manager/start completed in 19.15 seconds
2024-02-20 22:40:56,393 INFO    [dr2/0] Running addons/rook-operator/start
2024-02-20 22:40:56,562 INFO    [hub/1] addons/submariner/test completed in 46.58 seconds
2024-02-20 22:41:00,016 INFO    [dr2/2] addons/csi-addons/start completed in 22.77 seconds
2024-02-20 22:41:00,016 INFO    [dr2/2] Running addons/olm/start
2024-02-20 22:41:18,115 INFO    [hub/0] addons/olm/start completed in 28.65 seconds
2024-02-20 22:41:24,847 INFO    [dr1/2] addons/velero/start completed in 43.55 seconds
2024-02-20 22:41:24,847 INFO    [dr1/2] Running addons/velero/test
2024-02-20 22:41:40,656 INFO    [dr2/2] addons/olm/start completed in 40.64 seconds
2024-02-20 22:41:40,656 INFO    [dr2/2] Running addons/minio/start
2024-02-20 22:41:56,751 INFO    [dr1/1] addons/ocm-cluster/start completed in 148.06 seconds
2024-02-20 22:41:56,751 INFO    [dr1/1] Running addons/ocm-cluster/test
2024-02-20 22:41:58,669 INFO    [dr2/2] addons/minio/start completed in 18.01 seconds
2024-02-20 22:41:58,669 INFO    [dr2/2] Running addons/velero/start
2024-02-20 22:42:00,808 INFO    [dr1/2] addons/velero/test completed in 35.96 seconds
2024-02-20 22:42:04,079 INFO    [dr1/1] addons/ocm-cluster/test completed in 7.33 seconds
2024-02-20 22:42:04,079 INFO    [dr1/1] Running addons/recipe/start
2024-02-20 22:42:05,524 INFO    [dr1/1] addons/recipe/start completed in 1.45 seconds
2024-02-20 22:42:09,151 INFO    [dr2/0] addons/rook-operator/start completed in 72.76 seconds
2024-02-20 22:42:09,151 INFO    [dr2/0] Running addons/rook-cluster/start
2024-02-20 22:42:14,482 INFO    [dr2/1] addons/ocm-cluster/start completed in 97.23 seconds
2024-02-20 22:42:14,483 INFO    [dr2/1] Running addons/ocm-cluster/test
2024-02-20 22:42:21,355 INFO    [dr2/1] addons/ocm-cluster/test completed in 6.87 seconds
2024-02-20 22:42:21,355 INFO    [dr2/1] Running addons/recipe/start
2024-02-20 22:42:23,769 INFO    [dr2/1] addons/recipe/start completed in 2.41 seconds
2024-02-20 22:42:28,877 INFO    [dr2/2] addons/velero/start completed in 30.21 seconds
2024-02-20 22:42:28,877 INFO    [dr2/2] Running addons/velero/test
2024-02-20 22:43:03,706 INFO    [dr2/2] addons/velero/test completed in 34.83 seconds
2024-02-20 22:43:19,945 INFO    [dr1/0] addons/rook-cluster/start completed in 155.61 seconds
2024-02-20 22:43:19,945 INFO    [dr1/0] Running addons/rook-pool/start
2024-02-20 22:43:37,179 INFO    [dr1/0] addons/rook-pool/start completed in 17.23 seconds
2024-02-20 22:43:37,180 INFO    [dr1/0] Running addons/rook-toolbox/start
2024-02-20 22:43:40,974 INFO    [dr1/0] addons/rook-toolbox/start completed in 3.79 seconds
2024-02-20 22:44:17,988 INFO    [dr2/0] addons/rook-cluster/start completed in 128.84 seconds
2024-02-20 22:44:17,988 INFO    [dr2/0] Running addons/rook-pool/start
2024-02-20 22:44:55,322 INFO    [dr2/0] addons/rook-pool/start completed in 37.33 seconds
2024-02-20 22:44:55,322 INFO    [dr2/0] Running addons/rook-toolbox/start
2024-02-20 22:44:58,185 INFO    [dr2/0] addons/rook-toolbox/start completed in 2.86 seconds
2024-02-20 22:44:58,185 INFO    [rdr/0] Running addons/rbd-mirror/start
2024-02-20 22:44:58,186 INFO    [rdr/1] Running addons/volsync/start
2024-02-20 22:45:31,319 INFO    [rdr/1] addons/volsync/start completed in 33.13 seconds
2024-02-20 22:45:31,319 INFO    [rdr/1] Running addons/volsync/test
2024-02-20 22:45:57,297 ERROR   [rdr/0] Cluster failed
Traceback (most recent call last):
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 129, in execute
    f.result()
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 289, in run_worker
    run_addon(addon, worker["name"], hooks=hooks, allow_failure=allow_failure)
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 305, in run_addon
    run_hook(hook, addon["args"], name, allow_failure=allow_failure)
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 312, in run_hook
    run(hook, *args, name=name)
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 327, in run
    for line in commands.watch(*cmd):
  File "/home/nsoffer/src/ramen/test/drenv/commands.py", line 148, in watch
    raise Error(args, error, exitcode=p.returncode)
drenv.commands.Error: Command failed:
   command: ('addons/rbd-mirror/start', 'dr1', 'dr2')
   exitcode: 1
   error:
      Traceback (most recent call last):
        File "/home/nsoffer/src/ramen/test/addons/rbd-mirror/start", line 175, in <module>
          check_blocklist(cluster2)
        File "/home/nsoffer/src/ramen/test/addons/rbd-mirror/start", line 37, in check_blocklist
          raise RuntimeError(f"Ceph blocklist on cluster {cluster}: {blocklist}")
      RuntimeError: Ceph blocklist on cluster dr2: [{'addr': '192.168.122.192:0/216466187', 'until': '2024-02-20T21:45:03.442684+0000'}]

2024-02-20 22:46:31,992 INFO    [rdr/1] addons/volsync/test completed in 60.67 seconds

got it, I will check the failure and see what changes are required.

@yati1998 yati1998 changed the title update rook version from 12 - 13 [DNM]update rook version from 12 - 13 Feb 21, 2024
@nirs
Copy link
Member

nirs commented Feb 21, 2024

@yati1998 we need to understand better why new osd blocklist are created - so far we
just cleared them before setting up rbd-mirror, and check that we don't have new
entries before testing rbd mirroring. We know that if we leave the blocklist we
have issues later when testing ramen.

I will run more tests locally, we must have this change work on volume snapshot replication.

@nirs
Copy link
Member

nirs commented Feb 21, 2024

Next run timed out when waiting for the pool to become healthy.

I run successful tests today with regional-dr-kubevirt.yaml and rook 1.12, so upgrading
to 1.13 may need more work.

$ drenv start envs/regional-dr.yaml 
2024-02-21 18:27:35,422 INFO    [rdr] Starting environment
2024-02-21 18:27:35,543 INFO    [dr1] Starting minikube cluster
2024-02-21 18:27:36,513 INFO    [dr2] Starting minikube cluster
2024-02-21 18:27:37,496 INFO    [hub] Starting minikube cluster
2024-02-21 18:31:11,184 INFO    [hub] Cluster started in 213.69 seconds
2024-02-21 18:31:11,185 INFO    [hub/0] Running addons/ocm-hub/start
2024-02-21 18:31:11,185 INFO    [hub/1] Running addons/submariner/start
2024-02-21 18:31:59,125 INFO    [hub/0] addons/ocm-hub/start completed in 47.94 seconds
2024-02-21 18:31:59,125 INFO    [hub/0] Running addons/ocm-controller/start
2024-02-21 18:32:17,419 INFO    [hub/0] addons/ocm-controller/start completed in 18.29 seconds
2024-02-21 18:32:17,419 INFO    [hub/0] Running addons/cert-manager/start
2024-02-21 18:32:24,102 INFO    [hub/1] addons/submariner/start completed in 72.92 seconds
2024-02-21 18:32:24,102 INFO    [hub/1] Running addons/submariner/test
2024-02-21 18:32:29,767 INFO    [dr1] Cluster started in 294.22 seconds
2024-02-21 18:32:29,768 INFO    [dr1/0] Running addons/cert-manager/start
2024-02-21 18:32:29,768 INFO    [dr1/1] Running addons/ocm-cluster/start
2024-02-21 18:32:29,770 INFO    [dr1/2] Running addons/csi-addons/start
2024-02-21 18:32:32,271 INFO    [hub/0] addons/cert-manager/start completed in 14.85 seconds
2024-02-21 18:32:32,271 INFO    [hub/0] Running addons/olm/start
2024-02-21 18:32:49,334 INFO    [dr1/0] addons/cert-manager/start completed in 19.57 seconds
2024-02-21 18:32:49,334 INFO    [dr1/0] Running addons/rook-operator/start
2024-02-21 18:32:53,458 INFO    [dr1/2] addons/csi-addons/start completed in 23.69 seconds
2024-02-21 18:32:53,458 INFO    [dr1/2] Running addons/olm/start
2024-02-21 18:32:56,285 INFO    [hub/0] addons/olm/start completed in 24.01 seconds
2024-02-21 18:33:28,617 INFO    [hub/1] addons/submariner/test completed in 64.51 seconds
2024-02-21 18:33:31,335 INFO    [dr1/0] addons/rook-operator/start completed in 42.00 seconds
2024-02-21 18:33:31,335 INFO    [dr1/0] Running addons/rook-cluster/start
2024-02-21 18:33:46,866 INFO    [dr1/2] addons/olm/start completed in 53.41 seconds
2024-02-21 18:33:46,866 INFO    [dr1/2] Running addons/minio/start
2024-02-21 18:34:03,797 INFO    [dr1/1] addons/ocm-cluster/start completed in 94.03 seconds
2024-02-21 18:34:03,797 INFO    [dr1/1] Running addons/ocm-cluster/test
2024-02-21 18:34:05,668 INFO    [dr2] Cluster started in 389.15 seconds
2024-02-21 18:34:05,668 INFO    [dr2/0] Running addons/cert-manager/start
2024-02-21 18:34:05,668 INFO    [dr2/1] Running addons/ocm-cluster/start
2024-02-21 18:34:05,669 INFO    [dr2/2] Running addons/csi-addons/start
2024-02-21 18:34:11,356 INFO    [dr1/2] addons/minio/start completed in 24.49 seconds
2024-02-21 18:34:11,356 INFO    [dr1/2] Running addons/velero/start
2024-02-21 18:34:13,805 INFO    [dr1/1] addons/ocm-cluster/test completed in 10.01 seconds
2024-02-21 18:34:13,805 INFO    [dr1/1] Running addons/recipe/start
2024-02-21 18:34:16,870 INFO    [dr1/1] addons/recipe/start completed in 3.06 seconds
2024-02-21 18:34:24,621 INFO    [dr2/0] addons/cert-manager/start completed in 18.95 seconds
2024-02-21 18:34:24,621 INFO    [dr2/0] Running addons/rook-operator/start
2024-02-21 18:34:28,909 INFO    [dr2/2] addons/csi-addons/start completed in 23.24 seconds
2024-02-21 18:34:28,909 INFO    [dr2/2] Running addons/olm/start
2024-02-21 18:34:49,810 INFO    [dr1/2] addons/velero/start completed in 38.45 seconds
2024-02-21 18:34:49,811 INFO    [dr1/2] Running addons/velero/test
2024-02-21 18:34:55,865 INFO    [dr2/2] addons/olm/start completed in 26.96 seconds
2024-02-21 18:34:55,865 INFO    [dr2/2] Running addons/minio/start
2024-02-21 18:35:17,120 INFO    [dr2/2] addons/minio/start completed in 21.26 seconds
2024-02-21 18:35:17,120 INFO    [dr2/2] Running addons/velero/start
2024-02-21 18:35:22,909 INFO    [dr1/2] addons/velero/test completed in 33.10 seconds
2024-02-21 18:35:26,442 INFO    [dr2/0] addons/rook-operator/start completed in 61.82 seconds
2024-02-21 18:35:26,442 INFO    [dr2/0] Running addons/rook-cluster/start
2024-02-21 18:35:44,932 INFO    [dr1/0] addons/rook-cluster/start completed in 133.60 seconds
2024-02-21 18:35:44,932 INFO    [dr1/0] Running addons/rook-pool/start
2024-02-21 18:35:50,435 INFO    [dr2/1] addons/ocm-cluster/start completed in 104.77 seconds
2024-02-21 18:35:50,435 INFO    [dr2/1] Running addons/ocm-cluster/test
2024-02-21 18:35:56,701 INFO    [dr2/1] addons/ocm-cluster/test completed in 6.27 seconds
2024-02-21 18:35:56,701 INFO    [dr2/1] Running addons/recipe/start
2024-02-21 18:35:57,967 INFO    [dr2/1] addons/recipe/start completed in 1.27 seconds
2024-02-21 18:35:58,556 INFO    [dr2/2] addons/velero/start completed in 41.44 seconds
2024-02-21 18:35:58,556 INFO    [dr2/2] Running addons/velero/test
2024-02-21 18:36:30,095 INFO    [dr1/0] addons/rook-pool/start completed in 45.16 seconds
2024-02-21 18:36:30,095 INFO    [dr1/0] Running addons/rook-toolbox/start
2024-02-21 18:36:33,561 INFO    [dr2/2] addons/velero/test completed in 35.00 seconds
2024-02-21 18:36:33,839 INFO    [dr1/0] addons/rook-toolbox/start completed in 3.74 seconds
2024-02-21 18:37:30,754 INFO    [dr2/0] addons/rook-cluster/start completed in 124.31 seconds
2024-02-21 18:37:30,755 INFO    [dr2/0] Running addons/rook-pool/start
2024-02-21 18:37:52,924 INFO    [dr2/0] addons/rook-pool/start completed in 22.17 seconds
2024-02-21 18:37:52,924 INFO    [dr2/0] Running addons/rook-toolbox/start
2024-02-21 18:37:59,270 INFO    [dr2/0] addons/rook-toolbox/start completed in 6.35 seconds
2024-02-21 18:37:59,271 INFO    [rdr/0] Running addons/rbd-mirror/start
2024-02-21 18:37:59,271 INFO    [rdr/1] Running addons/volsync/start
2024-02-21 18:38:32,294 INFO    [rdr/1] addons/volsync/start completed in 33.02 seconds
2024-02-21 18:38:32,294 INFO    [rdr/1] Running addons/volsync/test
2024-02-21 18:39:13,492 INFO    [rdr/1] addons/volsync/test completed in 41.20 seconds
2024-02-21 18:43:08,131 ERROR   [rdr/0] Cluster failed
Traceback (most recent call last):
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 129, in execute
    f.result()
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 289, in run_worker
    run_addon(addon, worker["name"], hooks=hooks, allow_failure=allow_failure)
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 305, in run_addon
    run_hook(hook, addon["args"], name, allow_failure=allow_failure)
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 312, in run_hook
    run(hook, *args, name=name)
  File "/home/nsoffer/src/ramen/test/drenv/__main__.py", line 327, in run
    for line in commands.watch(*cmd):
  File "/home/nsoffer/src/ramen/test/drenv/commands.py", line 148, in watch
    raise Error(args, error, exitcode=p.returncode)
drenv.commands.Error: Command failed:
   command: ('addons/rbd-mirror/start', 'dr1', 'dr2')
   exitcode: 1
   error:
      Traceback (most recent call last):
        File "/home/nsoffer/src/ramen/test/addons/rbd-mirror/start", line 168, in <module>
          wait_until_pool_mirroring_is_healthy(cluster1)
        File "/home/nsoffer/src/ramen/test/addons/rbd-mirror/start", line 109, in wait_until_pool_mirroring_is_healthy
          kubectl.wait(
        File "/home/nsoffer/src/ramen/test/drenv/kubectl.py", line 129, in wait
          _watch("wait", *args, context=context, log=log)
        File "/home/nsoffer/src/ramen/test/drenv/kubectl.py", line 145, in _watch
          for line in commands.watch(*cmd, input=input):
        File "/home/nsoffer/src/ramen/test/drenv/commands.py", line 148, in watch
          raise Error(args, error, exitcode=p.returncode)
      drenv.commands.Error: Command failed:
         command: ('kubectl', 'wait', '--context', 'dr1', 'cephblockpools.ceph.rook.io', 'replicapool', '--for=jsonpath={.status.mirroringStatus.summary.daemon_health}=OK', '--namespace=rook-ceph', '--timeout=300s')
         exitcode: 1
         error:
            error: timed out waiting for the condition on cephblockpools/replicapool

@nirs
Copy link
Member

nirs commented Feb 21, 2024

Running with rook.yaml works fine - this does the same work we do in regional-dr
including the rbd-mirror/test self test.

Maybe the failures are related to overloading the laptop. From my experience we this
happens more when my laptop over heating because the air filters are clogged.

$ drenv start envs/rook.yaml 
2024-02-21 18:46:49,216 INFO    [rook] Starting environment
2024-02-21 18:46:49,267 INFO    [dr1] Starting minikube cluster
2024-02-21 18:46:50,306 INFO    [dr2] Starting minikube cluster
2024-02-21 18:49:13,516 INFO    [dr2] Cluster started in 143.21 seconds
2024-02-21 18:49:13,516 INFO    [dr2/0] Running addons/rook-operator/start
2024-02-21 18:49:13,516 INFO    [dr2/1] Running addons/csi-addons/start
2024-02-21 18:49:45,460 INFO    [dr2/1] addons/csi-addons/start completed in 31.94 seconds
2024-02-21 18:49:52,665 INFO    [dr1] Cluster started in 183.40 seconds
2024-02-21 18:49:52,666 INFO    [dr1/0] Running addons/rook-operator/start
2024-02-21 18:49:52,666 INFO    [dr1/1] Running addons/csi-addons/start
2024-02-21 18:49:52,785 INFO    [dr2/0] addons/rook-operator/start completed in 39.27 seconds
2024-02-21 18:49:52,785 INFO    [dr2/0] Running addons/rook-cluster/start
2024-02-21 18:50:24,720 INFO    [dr1/1] addons/csi-addons/start completed in 32.05 seconds
2024-02-21 18:50:28,913 INFO    [dr1/0] addons/rook-operator/start completed in 36.25 seconds
2024-02-21 18:50:28,913 INFO    [dr1/0] Running addons/rook-cluster/start
2024-02-21 18:51:35,819 INFO    [dr2/0] addons/rook-cluster/start completed in 103.03 seconds
2024-02-21 18:51:35,819 INFO    [dr2/0] Running addons/rook-pool/start
2024-02-21 18:52:05,692 INFO    [dr2/0] addons/rook-pool/start completed in 29.87 seconds
2024-02-21 18:52:05,693 INFO    [dr2/0] Running addons/rook-toolbox/start
2024-02-21 18:52:08,394 INFO    [dr2/0] addons/rook-toolbox/start completed in 2.70 seconds
2024-02-21 18:52:14,360 INFO    [dr1/0] addons/rook-cluster/start completed in 105.45 seconds
2024-02-21 18:52:14,360 INFO    [dr1/0] Running addons/rook-pool/start
2024-02-21 18:52:48,415 INFO    [dr1/0] addons/rook-pool/start completed in 34.05 seconds
2024-02-21 18:52:48,415 INFO    [dr1/0] Running addons/rook-toolbox/start
2024-02-21 18:52:50,715 INFO    [dr1/0] addons/rook-toolbox/start completed in 2.30 seconds
2024-02-21 18:52:50,715 INFO    [rook/0] Running addons/rbd-mirror/start
2024-02-21 18:53:50,348 INFO    [rook/0] addons/rbd-mirror/start completed in 59.63 seconds
2024-02-21 18:53:50,348 INFO    [rook/0] Running addons/rbd-mirror/test
2024-02-21 18:54:03,232 INFO    [rook/0] addons/rbd-mirror/test completed in 12.88 seconds
2024-02-21 18:54:03,232 INFO    [rook] Environment started in 434.02 seconds

@nirs
Copy link
Member

nirs commented Feb 21, 2024

Did a successful run with regional-dr.yaml and run basic-test. I think we can
merge this - the issues are not with rook but with the environment.

Regardless we should understand why we see blocklist entries and why
the are created again while configuring rbd mirroring.

Tested on top of #1153

$ basic-test/run -c configs/k8s/busybox-regional-rbd-deploy.yaml envs/regional-dr.yaml 
2024-02-21 19:51:51,644 INFO    [deploy] Deploying application
2024-02-21 19:51:51,644 INFO    [deploy] Deploying drpolicy 'busybox-regional-rbd-deploy'
2024-02-21 19:51:51,815 INFO    [deploy] Waiting until drpolicy 'busybox-regional-rbd-deploy' is validated
2024-02-21 19:51:51,972 INFO    [deploy] Deploying application 'busybox-regional-rbd-deploy'
2024-02-21 19:51:53,398 INFO    [deploy] Waiting for 'placement.cluster.open-cluster-management.io/busybox-placement' decisions
2024-02-21 19:51:53,616 INFO    [deploy] Application running on cluster 'dr1'
2024-02-21 19:51:53,724 INFO    [enable-dr] Enable DR
2024-02-21 19:51:53,774 INFO    [enable-dr] Disabling OCM scheduling for 'placement.cluster.open-cluster-management.io/busybox-placement'
2024-02-21 19:51:53,890 INFO    [enable-dr] Waiting for 'placement.cluster.open-cluster-management.io/busybox-placement' decisions
2024-02-21 19:51:54,240 INFO    [enable-dr] waiting for namespace busybox-regional-rbd-deploy
2024-02-21 19:51:54,357 INFO    [enable-dr] Waiting until 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' reports status
2024-02-21 19:51:54,700 INFO    [enable-dr] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' Available condition
2024-02-21 19:51:54,877 INFO    [enable-dr] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' PeerReady condition
2024-02-21 19:51:55,037 INFO    [enable-dr] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' first replication
2024-02-21 19:53:24,150 INFO    [enable-dr] DR enabled
2024-02-21 19:53:24,280 INFO    [failover] Fail over application
2024-02-21 19:53:24,342 INFO    [failover] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' Available condition
2024-02-21 19:53:24,528 INFO    [failover] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' PeerReady condition
2024-02-21 19:53:24,712 INFO    [failover] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' first replication
2024-02-21 19:53:24,862 INFO    [failover] Waiting for 'placement.cluster.open-cluster-management.io/busybox-placement' decisions
2024-02-21 19:53:25,151 INFO    [failover] Starting failover for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' to cluster 'dr2'
2024-02-21 19:53:25,320 INFO    [failover] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' Available condition
2024-02-21 19:53:54,311 INFO    [failover] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' PeerReady condition
2024-02-21 19:56:24,169 INFO    [failover] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' first replication
2024-02-21 19:56:24,260 INFO    [failover] Application was failed over
2024-02-21 19:56:24,371 INFO    [relocate] Relocate application
2024-02-21 19:56:24,416 INFO    [relocate] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' Available condition
2024-02-21 19:56:24,569 INFO    [relocate] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' PeerReady condition
2024-02-21 19:56:24,722 INFO    [relocate] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' first replication
2024-02-21 19:56:24,817 INFO    [relocate] Waiting for 'placement.cluster.open-cluster-management.io/busybox-placement' decisions
2024-02-21 19:56:25,101 INFO    [relocate] Starting relocate for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' to cluster 'dr1'
2024-02-21 19:56:25,243 INFO    [relocate] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' phase 'Relocated'
2024-02-21 19:58:54,200 INFO    [relocate] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' Available condition
2024-02-21 19:58:54,366 INFO    [relocate] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' PeerReady condition
2024-02-21 19:59:24,207 INFO    [relocate] Waiting for 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc' first replication
2024-02-21 19:59:24,275 INFO    [relocate] Application was relocated
2024-02-21 19:59:24,411 INFO    [disable-dr] Disable DR
2024-02-21 19:59:24,467 INFO    [disable-dr] Deleting 'drplacementcontrol.ramendr.openshift.io/busybox-regional-rbd-deploy-drpc'
2024-02-21 19:59:54,134 INFO    [disable-dr] Enabling OCM scheduling for 'placement.cluster.open-cluster-management.io/busybox-placement'
2024-02-21 19:59:54,210 INFO    [disable-dr] DR was disabled
2024-02-21 19:59:54,361 INFO    [undeploy] Deleting application
2024-02-21 19:59:54,361 INFO    [undeploy] Undeploying application 'busybox-regional-rbd-deploy'
2024-02-21 20:00:00,525 INFO    [undeploy] Deleting drpolicy 'busybox-regional-rbd-deploy'
2024-02-21 20:00:00,679 INFO    [undeploy] Application was deleted

@nirs nirs changed the title [DNM]update rook version from 12 - 13 Update rook version from 1.12 to 1.13 Feb 22, 2024
@nirs nirs merged commit 6f43ea3 into RamenDR:main Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nirs nirs nirs approved these changes

@raghavendra-talur raghavendra-talur raghavendra-talur approved these changes

+1 more reviewer

@Shwetha-Acharya Shwetha-Acharya Shwetha-Acharya approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

update rook version to v1.13

4 participants

@yati1998 @nirs @raghavendra-talur @Shwetha-Acharya